Recursive filtering a video image using activity mapping

ABSTRACT

A method of recursive filtering a video image includes determining a local variance σ 2  in luminosity in the vicinity of a picture element on the image; making an estimate of a value of noise variance σ r   2 ; obtaining a surface for value of a proportional parameter K from the equation 
             K   =       α   ⁡     [       ρ   ·   λ   ·     σ   2           ρ   ·   λ   ·     σ   2       +     τ   ·     σ   r   2           ]       +   β           
where 1≦σ≦128, 1 ≦σ r ≦64 and ρ, τ, α, and β are empirical constants, and selecting a value of λ to scale a relative contribution to the value of K by the luminosity variance and the noise variance. The image is recursively filtered using the proportional parameter K to sum proportions of the current image and an immediately preceding image.

FIELD OF THE INVENTION

This invention relates to a method of recursive filtering a video imageusing activity mapping.

BACKGROUND OF THE INVENTION

It is known, from, for example BBC Research Department Report “Videonoise reduction” BBC RD 1984/7 N. E. Tanton et al, July 1984, (Tanton)that random noise in a sequence of television or some other kinds ofelectronically generated images, e.g. scanned film, can be reduced byapplying a recursive temporal filter which operates on each pictureelement, hereinafter abbreviated to ‘pel’. It is beneficial to reducenoise levels prior to viewing images but also prior to processes thatare sensitive to the presence of noise, especially compression systemssuch as those defined by, but not limited to, MPEG specifications. Inpractice, noise control is among several important and valuabletechniques employed in pre-processing elements inherent in moderncompression hardware and software realisations of specifications such asMPEG.

For each pel, indexed in the image by i, an output R(i) of a recursivefilter is a sum of complementary proportions of a current image C(i) anda previous resultant output image S(i) such that the proportions of C(i)and S(i) in the output R(i) are controlled by means of a fractionalparameter K. If this notation is extended with a suffix ‘f’ to denotethe frame it is evident that the condition S_(f)(i)=R_(f−1)(i) ensures afirst order recursive temporal filter operation.

Hence, each pel in the result R(i) is expressed as:R _(f)(i)=K·C _(f)(i)+(1−K)·S _(f−1)(i) 0≦K≦1   Equation 1: Recursivenoise reduction calculation

Where:

C_(f)(i) is the current input image under operation; i is the index toindividual pels of that image and f is the index to the sequence ofcomplete images,

S_(f)(i) is a stored array of pels equal to the result R_(f−1)(i) of thefilter operation on the previous image and i is the index to individualpels of that image.

R(i) is the resulting pel field, which, after the operation, is copiedto S(i) before the operation is performed on the next image in thesequence and i is the index to individual pels of that image.

It is expected in the filter calculations that the index i of each ofR(i), C(i) and S(i) is the same so that the pels are spatiallyco-located in each respective image. The index of pels in each image, i,may be defined in more than one-dimension; conventionally it isconvenient to use two, corresponding to the commonly-used rasterscanning of television images. Thus R_(f)(i) becomes R_(f)(x,y). Theindex f corresponds to the frame or temporal dimension.

K is a fractional parameter that controls a proportion of the currentimage C(x,y) to previous image S(x,y) used to obtain the current resultR(x,y). In the prior art, the parameter K does not change frequentlywith time, typically once per complete image, and in extreme cases maybe fixed indefinitely at one value. The fractional parameter may also beoperated under direct manual control. Experience shows that this is notsatisfactory for some image material.

The value of K is used to control the degree of filtering (and hencenoise attenuation) and that attenuation increases as K tends toward 0.At value K=0 there is no transmission of the incoming image C and theoutput is therefore “frozen”. This is an ideal setting to process stillimages where the absence of motion allows optimal filtering for noisereduction. In the presence of motion, however, the setting of K=O is, ingeneral, far from ideal. At K=1 there is no filtering at all and theoutput is identically equal to the input and no noise reduction isachieved.

This technique has some limitations. Objects in motion captured insuccessive images cause their respective pels to change position in theimage and thus their index (x,y). Motion leads to changes in theposition, i.e. lateral movement, or attitude, i.e. rotational movement,of recognisable objects in the image even when there is no cameramovement. Changes in object position are also caused by camera movementsuch as panning or tilting, or both, and changes in object position andscale caused by zooming, i.e. changes in focal length of the cameralens. In all these cases the recursive filter described above willtherefore add together pels in successive images whose indices (x,y) nolonger correspond to the same part of such moving objects andconsequently motion will be confused with noise and the filter willreduce the value of the differences between successive images caused bymotion as if it were noise. As a result, viewed images tend to blur forsmall amounts of movement and larger motion tends to create ghosts ofany moving objects substantially compromising the image with artefacts.The essence of the problem is an inability to discriminate reliablybetween noise and motion, in that small amounts of motion can bedifficult to distinguish from noise.

To reduce unwanted artefacts, the value of K can be defined on a per pelbasis, i.e. K(x,y), from information derived from the pel C(x,y) underoperation and the surrounding pels. The value of K(x,y) can be derivedfrom the temporal difference between co-located pels, however thisproduces artefacts because this local difference is a function both ofthe sought after motion and the noise in the image. To discriminatebetween noise and motion requires the measurement and the subsequentexploitation of differing statistical properties of the motion and thenoise.

Reinhard Kiette & Piero Zamperoni: Handbook of image processingoperators. John Wiley & Sons Chichester, UK. August 1996. ISBN 0 47195642 2 (Klette and Zamperoni) proposes a method for reducing noise in asingle static image by operating dynamically across the image andproducing an output image from a fraction of the “local average” ofluminance pel intensity and the intensity of the collocated pel.

The fraction of the local average used is set by a function of what theydescribe as the “local variance”, σ. In other words the degree offiltering is determined by the amount of local activity in the image. Inareas where there is such activity the degree of filtering is reducedbecause noise is less visible than in inactive areas where visibility isgreater.

Klette and Zamperoni define their operators as follows:

$\begin{matrix}{{Calculation}\mspace{14mu}{of}\mspace{14mu}{local}\mspace{14mu}{average}} & \; \\{\left. {\overset{\_}{F}\left( {r,s} \right)} \right) = {\frac{1}{a}{\sum\limits_{i = {- m}}^{m}{\sum\limits_{j = {- n}}^{n}{f\left( {{r - i},{s - j}} \right)}}}}} & {{Equation}\mspace{14mu} 2}\end{matrix}$

This defines a local average of luminance intensity around a pel atindex position (r,s) as the sum of luminance pels in an arbitrary windowof 2m+1 by 2n+1 pels divided by ‘a’, the number of pels in the windowand a=(2m+1).(2n+1).

Klette and Zamperoni also define σ(r,s), the local variance i.e. themean of the squares of pel values, as:

$\begin{matrix}{{Calculation}\mspace{14mu}{of}\mspace{14mu}{local}\mspace{14mu}{variance}} & \; \\{{\sigma\left( {r,s} \right)} = {{\overset{\sim}{F}\left( {r,s} \right)} = {\frac{1}{a}{\sum\limits_{i = {- m}}^{m}{\sum\limits_{j = {- n}}^{n}\left\lbrack {{f\left( {{r - i},{s - j}} \right)} - {\overset{\_}{F}\left( {r,s} \right)}} \right\rbrack^{2}}}}}} & {{Equation}\mspace{14mu} 3}\end{matrix}$

The operation they describe reduces noise by softening the image towardthe local mean of luminance values but this causes blurring andsoftening as artefacts.

Klette and Zamperoni further describe that the degree of softeningapplied to a pel (K) is defined by,

$\begin{matrix}{{Calculation}\mspace{14mu}{of}\mspace{14mu}{applied}\mspace{14mu}{degree}\mspace{14mu}{of}\mspace{14mu}{filtering}} & \; \\{K = \frac{\sigma^{2}}{\sigma^{2} + \sigma_{r}^{2}}} & {{Equation}\mspace{14mu} 4}\end{matrix}$

Where σ is a measured local variance from mean luminance level and σ_(r)is an estimated value of the noise variance.

Expressed graphically as a surface, the calculation from Equation 4 isas illustrated in FIG. 1.

It is an object of the present invention at least to ameliorate theaforesaid shortcomings in the prior art.

It is another object of the present invention to provide means wherebydiscrimination between motion and noise may be improved.

It is a further object of this invention that the value of K may bevaried pel-by-pel appropriately to reduce noise but also to suppressunwanted artefacts, so that K becomes K(x,y).

SUMMARY OF THE INVENTION

According to a first aspect of the present invention there is provided amethod of recursive filtering a video image comprising the steps of:determining a local variance σ² in luminosity in the vicinity of apicture element on the image; selecting an estimated value of noisevariance σ_(r) ² indicative of characteristics of that noise; obtaininga surface for value of a proportional parameter K from an equation

${K = {{\alpha\left\lbrack \frac{\rho \cdot \lambda \cdot \sigma^{2}}{{\rho \cdot \lambda \cdot \sigma^{2}} + {\tau \cdot \sigma_{r}^{2}}} \right\rbrack} + \beta}},$where 1≦σ≦128, 1≦σ_(r)≦64 and ρ, τ, α and β are empirical constants;selecting a value of λ to scale a relative contribution to the value ofK by the luminosity variance and the noise variance; and recursivelyfiltering the image using the proportional parameter K to sumproportions of the current image and an immediately preceding image.

Conveniently, determining the local variance σ² in luminosity comprisesdetermining a mean of squares of differences between picture elementsvalues and a mean value of the luminosities of the picture elements in apredetermined vicinity of a picture element being processed.

Alternatively, determining the local variance σ² in luminosity comprisesdetermining a positive root of a sum of squares of differences betweenpicture elements values and a mean value of the luminosities of thepicture elements in a predetermined vicinity of a picture element beingprocessed.

Advantageously, the method further comprises selecting attenuationcharacteristics dependent on levels of motion and noise present in theimage.

Conveniently, for an image containing substantially a same level ofmotion and noise, the method further comprises choosing a level ofmotion/noise measurement requiring a given degree of attenuation andusing the attenuation curve to attenuate the required amount at thechosen level and progressively less above that level.

Conveniently, for an image containing a high level of motion, the methodfurther comprises selecting a value of λ such that no filtering occursabove a determined degree of motion.

Conveniently, for an image containing infrequent high levels of noise,the method further comprises use a portion of the surface that is notnormalised along the σ_(r) axis so that some attenuation is appliedwhatever the level of motion in the image.

According to a second aspect of the invention, there is provided arecursive filter system for a video image comprising: calculating meansarranged to determine a local variance σ² in luminosity in the vicinityof a picture element on the image; input means for inputting an estimateof a value of noise variance σ_(r) ² indicative of characteristics ofthat noise; processing means arranged to obtain a surface for values ofa proportional parameter K from an equation

${K = {{\alpha\left\lbrack \frac{\rho \cdot \lambda \cdot \sigma^{2}}{{\rho \cdot \lambda \cdot \sigma^{2}} + {\tau \cdot \sigma_{r}^{2}}} \right\rbrack} + \beta}},$where 1≦σ≦128, 1≦σ_(r)≦64 and ρ, τ; α and β are empirical constants;selection means arranged to permit selection of a value of λ to scale arelative contribution to the value of the proportional parameter K bythe luminosity variance and the noise variance; and a recursive filterarranged to filter the image using the proportional parameter K to sumproportions of the current image and an immediately preceding image.

Conveniently, the calculating means is arranged to determine a mean ofsquares of differences between picture elements values and a mean valueof the luminosities of the picture elements in a predetermined vicinityof a picture element being processed. Alternatively, the calculatingmeans is arranged to determine the local variance σ² in luminosity bydetermining a positive root of a sum of squares of differences betweenpicture elements values and a mean value of the luminosities of thepicture elements in a predetermined vicinity of a picture element beingprocessed.

Conveniently, the recursive filter system further comprises input meansfor inputting attenuation characteristics selected dependent on levelsof motion and noise present in the image.

Conveniently, for an image containing substantially a same level ofmotion and noise, the recursive filter system further comprises inputmeans arranged to allow choice of a level of motion/noise measurementrequiring a given degree of attenuation and arranged to use theattenuation curve to attenuate the required amount at the chosen leveland progressively less above that level.

Conveniently, for an image containing a high level of motion, therecursive filter system comprises input means for inputting a value of λsuch that no filtering occurs above a determined degree of motion.

Conveniently, for an image containing infrequent high levels of noise,the recursive filter system comprises means for selecting a portion ofthe surface that is not normalised along the σ_(r) axis so that someattenuation is applied whatever the level of motion in the image.

According to a third aspect of the invention, there is provided acomputer program comprising code means for performing all the steps ofthe method described above when the program is run on one or morecomputers.

BRIEF DESCRIPTION OF THE DRAWINGS

The invention will now be described, by way of example, with referenceto the accompanying drawings in which:

FIG. 1 is a graph according to the prior art of a surface, from Equation4, representing a value of a proportional parameter used in recursivefiltering;

FIG. 2: is a graph of a surface, from Equation 5, representing a valueof a proportional parameter used in recursive filtering according to theinvention; and

FIG. 3 is a flowchart of a method according to the apparatus.

In the Figures like reference numerals denote like parts.

DETAILED DESCRIPTION OF EMBODIMENTS

The approach of Klette and Zamperoni is suitable for still images buthas the disadvantage of causing excessive localised softening and otherartefacts when applied to moving image sequences. Their approach alsofails to address the benefit to be gained by using the temporalredundancy available in a sequence of frames.

The principle of recursive filtering of Tanton is sometimes used inmotion video but even if the degree of filtering is varied it still maynot attenuate the noise effectively while leaving the motion unaffected.

For recursive filtering an image is stored, step 31, for comparison witha current image which is subsequently input, step 32.

By taking elements from Klette and Zamperoni and replacing the“localised variance from mean luminance level” with a measure of motion,a profile of attenuation can be found that is designed to act on thenoise more effectively and avoid introducing artefacts in the motion. Itis further advantageous to scale the degree of motion by a coefficient,λ, to make a best utilisation of the attenuation curves, as discussedherein below.

The prior art formula from Equation 4 is sub-optimal for processingmotion video for several reasons; first the surface deviates from unityas σ_(r) increases. This means that the degree of attenuation is greaterthan zero for large degrees of motion. Secondly it is inappropriate toallow K to tend to zero since this would result in significant visualimpairment to small motion and level changes. For better noiseattenuation it is also necessary to scale and select an appropriatesection of the surface.

Hence, for moving images, according to the invention, the formula fromEquation 4 is modified to be:

$\begin{matrix}{{Modified}\mspace{14mu}{calculation}\mspace{14mu}{of}\mspace{14mu}{applied}\mspace{14mu}{degree}\mspace{14mu}{of}\mspace{14mu}{filtering}\text{:}} & \; \\{K = {{\alpha\left\lbrack \frac{\rho \cdot \lambda \cdot \sigma^{2}}{{\rho \cdot \lambda \cdot \sigma^{2}} + {\tau \cdot \sigma_{r}^{2}}} \right\rbrack} + \beta}} & {{Equation}\mspace{14mu} 5}\end{matrix}$where

-   1≦σ≦128,-   1≦σ_(r)≦64    and ρ, τ, α and β are empirical constants.

Suitable values of the empirical constants are found to be ρ=0.4,τ=1,α=0.9 and β=0.1.

By altering the segment of the surface in use (by varying values of ρand τ) and normalising along the a axis, σ more suitable range of K canbe chosen. The values of ρ, τ, α and β given above are purely exemplary.Values are chosen to provide a full range of attenuation options overthe σ axis and σ_(r) axis and as such are chosen empirically.

Normalising the surface means that for larger σ values, K tends to 1,which prevents ghosting on larger motion.

The coefficients α and β scale the surface, bounding the maximumattenuation and preventing artefacts that might be unsightly.

In order to apply the equation, a local variance of luminosity in avicinity of a picture element being processed is determined, step 33, asan estimate of the degree of motion, and an estimate of noise varianceσ_(r) ² in the vicinity is made, step 34. It will be understood thatσ_(r) is not only an estimate of the noise variance but also anindication of characteristics of that noise. A choice is made of anattenuation profile from those available, based on characteristics ofthe noise present. If the axis numbered from 1 to 128 in FIG. 2 ismapped to the estimated degree of motion then the axis labelled S1 toS64 is used to map σ_(r) (the chosen profile of attenuation). Thevertical axis ranging from 0 to 1 is K, where K=1 represents nofiltering applied to the image, so that a surface for values of K isobtained, step 35.

Optimal Use of Mapping Curves

Once this system is implemented the attenuation characteristics can bemanipulated to suit the image and the noise present.

Examples of such strategies are:

-   -   1. Choose, step 36, a level of motion/noise measurement        requiring a given degree of attenuation and use the attenuation        curve to attenuate, step 37, the chosen amount at that level and        to a progressively smaller degree above this to catch the        remainder of a wide noise distribution. This strategy is ideal        for images where the degree of motion and the noise floor        overlap due to either low motion and noise or higher motion and        high noise.    -   2. Choose a degree of motion above which filtering is not        desired and scale σ with λ so that this value sits at the        furthest end of the curve. This is useful for fast, large motion        such as sport that would otherwise suffer blurring from the        recursive algorithm. This method also draws on psycho-visual        models of the human visual system that reveal insensitivity to        noise on fast moving objects.    -   3. For material suffering large level noise on an infrequent        basis it would be appropriate to use a section of the curve that        is not normalised along the σ_(r) axis. This would mean that a        degree of attenuation would be applied no matter what the        measured motion is. This would apply to source images which have        suffered mechanically or chemically during storage resulting in        physical defects such as acetate flaws.

Alternative embodiments of the invention can be implemented as acomputer program product for use with a computer system, the computerprogram product being, for example, a series of computer instructionsstored on a tangible data recording medium, such as a diskette, CD-ROM,ROM, or fixed disk. The series of computer instructions can constituteall or part of the functionality described above, and can also be storedin any memory device, volatile or non-volatile, such as semiconductor,magnetic, optical or other memory device.

Although the present invention has been described with reference topreferred embodiments, workers skilled in the art will recognize thatchanges may be made in form and detail without departing from the spiritand scope of the invention.

I claim:
 1. A method of recursive filtering a video image comprising thesteps of: a. determining, using a computer system, a local variance σ²in luminosity in the vicinity of a picture element on the image; b.selecting, using the computer system, an estimated value of noisevariance σ_(r) ² indicative of characteristics of that noise; c.obtaining, using the computer system, a surface for value of aproportional parameter K from the equation$K = {{\alpha\left\lbrack \frac{\rho \cdot \lambda \cdot \sigma^{2}}{{\rho \cdot \lambda \cdot \sigma^{2}} + {\tau \cdot \sigma_{r}^{2}}} \right\rbrack} + \beta}$where 1≦σ≦128, 1≦σ_(r)≦64 and ρ, τ, α and β are empirical constants; d.electing, using the computer system, a value of λ to scale a relativecontribution to the value of the proportional parameter K by theluminosity variance and the noise variance; and e. recursivelyfiltering, using the computer system, the image using the proportionalparameter K to sum proportions of the current image and an immediatelypreceding image.
 2. A method as claimed in claim 1, wherein determiningthe local variance σ² in luminosity comprises determining, using thecomputer system, a mean of squares of differences between pictureelements values and a mean value of the luminosities of the pictureelements in a predetermined vicinity of a picture element beingprocessed.
 3. A method as claimed in claim 1, wherein determining thelocal variance σ² in luminosity comprises determining, using thecomputer system, a positive root of a sum of squares of differencesbetween picture elements values and a mean value of the luminosities ofthe picture elements in a predetermined vicinity of a picture elementbeing processed.
 4. A method as claimed in claim 1, further comprisingselecting, using the computer system, attenuation characteristicsdependent on levels of motion and noise present in the image.
 5. Amethod as claimed in claim 4, for an image containing a same level ofmotion and noise, further comprising choosing, using the computersystem, a level of motion/noise measurement requiring a given degree ofattenuation and using the attenuation curve to attenuate the requiredamount at the chosen level and progressively less above that level.
 6. Amethod as claimed in claim 4, for an image containing a high level ofmotion, comprising selecting, using the computer system, a value of λsuch that no filtering occurs above a determined degree of motion.
 7. Amethod as claimed in claim 4, for an image containing infrequent highlevels of noise, comprising using a portion of the surface that is notnormalised along the σ_(r) axis so that some attenuation is appliedwhatever the level of motion in the image.
 8. A recursive filter systemfor a video image comprising a non-transitory computer-readable storagemedium including computer-readable instructions, when executed by acomputer system, are configured to: a. determine a local variance σ² inluminosity in the vicinity of a picture element on the image; b. imputan estimate of a value of noise variance σ_(r) ² indicative ofcharacteristics of that noise; c. obtain a surface for value of aproportional parameter K from the equation$K = {{\alpha\left\lbrack \frac{\rho \cdot \lambda \cdot \sigma^{2}}{{\rho \cdot \lambda \cdot \sigma^{2}} + {\tau \cdot \sigma_{r}^{2}}} \right\rbrack} + \beta}$where ≦σ≦128, 1≦σ^(r)≦64 and ρ, τ, α and β are empirical constants; d.permit selection of a value of λto scale a relative contribution to thevalue of K by the luminosity variance and the noise variance; and e.filter the image using the proportional parameter K to sum proportionsof the current image and an immediately preceding image.
 9. A recursivefilter system as claimed in claim 8, wherein the computer-readableinstructions, when executed by the computer system, are furtherconfigured to determine a mean of squares of differences between pictureelements values and a mean value of the luminosities of the pictureelements in a predetermined vicinity of a picture element beingprocessed.
 10. A recursive filter system as claimed in claim 8, whereinthe computer-readable instructions, when executed by the computersystem, are further configured to determine the local variance σ² inluminosity by determining a positive root of a sum of squares ofdifferences between picture elements values and a mean value of theluminosities of the picture elements in a predetermined vicinity of apicture element being processed.
 11. A recursive filter system asclaimed in claim 10, for an image containing a same level of motion andnoise, further comprising means arranged to allow choice of a level ofmotion/noise measurement requiring a given degree of attenuation and touse the attenuation curve to attenuate the required amount at the chosenlevel and progressively less above that level.
 12. A recursive filtersystem as claimed in claim 10, for an image containing a high level ofmotion, the computer-readable instructions, when executed by thecomputer system, are further configured to input a value of λ such thatno filtering occurs above a determined degree of motion.
 13. A recursivefilter system as claimed in claim 10, for an image containing infrequenthigh levels of noise, the computer-readable instructions, when executedby the computer system, are further configured to select a portion ofthe surface that is not normalised along the σ_(r) axis so that someattenuation is applied whatever the level of motion in the image.
 14. Arecursive filter system as claimed in claim 8, wherein thecomputer-readable instructions, when executed by the computer system,are further configured to input attenuation characteristics selecteddependent on levels of motion and noise present in the image.